DOCUMENT RESUME 

ED 351 936 HE 025 997 



TITLE 



INSTITUTION 
PUB DATE 
NOTE 
PUB TYPE 



How Can Postsecondary Education Be More Accountable 
to Colorado Citizens? Colorado Commission on Higher 
Education Master Plan Background Paper* 
Colorado Commission on Higher Education, Denver* 
2A Sep 92 
21p. 

Reports - Evaluative/Feasibility (142) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOl Plus Postage. 

'^Accountability; College Outcomes Assessment; 
^Educational Assessment; ^Educational Quality; 
Evaluation Criteria; '^Higher Education; Long Range 
Planning; State Agencies; State Standards; ^Statewide 
Planning 

'^Colorado; State Initiatives 



ABSTRACT 

This paper explores current Colorado postsecondary 
academic assessment and accountability efforts and offers a framework 
for decision-making about how to alter or enhance those efforts. The 
paper makes a distinction between assessment and accountability 
efforts from the statewide perspective. Assessment is defined as the 
use of measurement techniques to determine the impact of education on 
students. Accountability is defined as the use of assessment results 
to inform the public of the performance of the state's postsecondary 
education system. Also described are current practices which range 
from the legislatively mandated Higher Education Accountability 
Program (HEAP) to institution-based activities. A following section 
analyzes the adequacy of these current efforts arguing that HEAP is 
generally successful, that program accreditation and program review 
are effective, and that "The Scorecard," an annual publication 
offering a summary of a number of measures, has potential for 
becoming a valuable tool. National perspectives on postsecondary 
assessment and accountability are also reviewed. The paper concludes 
by proposing four options: (1) maintain assessment and accountability 
efforts currently underway; (2) discontinue or scale back current 
assessment activities, while enhancing accountability; (3) develop 
one system to do both assessment and accountability; and (A) continue 
current assessment activities, while enhancing accountability 
efforts. (Author/ JB) 
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HOW CAN POSTSECONDARY EDUCATION BE 
MORE ACCOUNTABLE TO COLORADO CITIZENS? 

(COLORADO HIGHER EDUCATION MASTER PLAN ISSUE #4) 



EXECUTIVE SUMMARY 



Public perception of the competencies of college graduates, both in Colorado and nationally, 
has Increased interest in developing ways to assess the college competencies of graduates as 
well as increase the accountability of postsecondary education to the public. This paper 
provides a framework for the Colorado Commission on Higher Education to decide whether 
current postsecondary academic assessment and accountability efforts are sufHcient. 

The paper flrst makes a distinction between assessment ana accountability efforts from the 
statewide perspective, then describes and evaluates these current efforts* National 
perspectives on postsecondary assessment and accountability are introduced to assist the 
consideration of these efforts in Colorado* The paper concludes with four options for 
Commission consideration* They are: 

Option 1: Maintain assessment and accountability efforts that are currently underway. 



Option 2: Discontinue (or downsize) current assessment activities, while enhancing 
accountability efforts* 

Option 3: Develop one system to accomplish objectives of both assessment and 
accountability* 

. Option 4: Continue current assessment activities, while enhancing accountability efforts* 



This paper was prepared for Commission discussion by a CCHE staff and commissioner 
committee that included Mark Chisholm, Stephanie Cunningham, Commissioner Christine 
Johnson, Geri Reinardy, and Sharon Samson* 



ERIC 



1 

3 



1. INTRODUCTION 



The purpose of this paper is to develop an analytical framework for the Colorado 
Commission on Higher Education to use for two purposes* First, to evaluate whether 
current academic assessment efforts (includmg what is widely known as the statewide 
accountability program) at public postsecondary institutions are accomplishing their 
objectives. Second, to assist the Commission in determining to what extent individual 
Colorado institutions, as well as the entire Colorado system of public postsecondary 
education, should enhance accountability efforts— efforts that articulate high standards for 
postsecondary education, then measure and report on the achievement of these standards 
to the public* 



n, BACKGROUND 
Purpose 

There are several motivations for this project. The first is the public perception that 
colleges are not adequately preparing graduates to be both productive professionals and 
responsible citizens* Increasingly, some business leaders, employers, and to a certain 
extent, the general public, charge that college graduates do not have the technical or 
general information that they should* In addition, the media and popular press have 
promoted this notion* News stories compare how little American adults know about 
geography, history and other subjects compared to non-Americans* Widely read books 
such as The Closing of the American Mind * and Cultural Literacy * which declake how little 
Americans know, have captured the attention of many Americans, and have risen to the 
top of bestseller lists in recent years* The federal government has also taken an interest in 
this subject* A 1991 U.S. Department of Labor Report concluded that workers lack the 
basic skills needed in the workplace* In addition to all these factors, there is little doubt 
that recent concern about the competencies of public K-12 graduates has also affected the 
perception about college graduates* 

The accuracy of these public perceptions is uncertain* This uncertainty creates a dUenmia 
for American higher education* Because of the absence of an effective, systematic process 
to assess the abilities of its graduates, higher education can neither dispel or confirm this 
perception* As a result, interest m assessment and accountability is gaining momentum at 
the national and state levels* At the national level, three signiHcant activities are underway* 
Federal "Student Right to Know" legislation requires all colleges to report student 
graduation rates by 1993* Also, recently passed federal "Ability to Benefit" legislation 
requires first-time college students without a high school diploma who want to receive 
federal student aid to pass an entrance exam. In addition, a National Governors' 
Association task force, as part of President Bush's National Education Goals Program, has 



ERIC 



2 

4 



t 



recommended the development of a voluntary national postsecondary assessment of 
graduates' abiUty to think critically, communicate effectively, and solve problems. 

In Colorado, legislative interest iu assessing college students became apparent in 1985 with 
the passage of higher education accountability legislation. This legislation requires 
institutions of higher education to "design and implement a systematic program to assess 
the knowledge, capacities, and skills developed by students in academic and co-curricular 
programs." While the Colorado higher education accountability legislation was primarily 
designed to be "assessment" oriented, i.e, gathering information about the performance of 
students, it also established an expectation of "accountability" by requiring institutions to 
publicly disclose the results of student assessment. Specifically, the law requires the results 
of institutional assessment "shall be communicated to the public, its students, and potential 
students." 

Meaning of rnmm nnl y-TTsed Termg Throughout this Paper 

The following definitions, or conmiou understandings of selected terms, will be used 
throughout this paper. 

Assessment: Assessment is the use of measurement techniques to determine the 

impact of education on students. The pui ose of assessment is to 
improve student learning. The primary users of assessment results 
are faculty. 

Student assessment is outcome oriented, referring to activities that 
measure what students know and learn. One example is the Colorado 
Higher Education Accountability Program (HEAP). Institutional 
assessment is process oriented, determining if institutions have 
provided the services needed to facilitate student learning. Examples 
include academic program review, and accreditation review. 

Accountability: Accountability is the use of assessment results to inform the public of 
the performance of the state's postsecondary education system. The 
purpose of accountability efforts is to assure the public, and elected 
officials that represent them, that tax dollars invested in postsecondary 
education are used eflectively. Accountability efforts can also assist 
state-level policy makers in their responsibilities. Since the primary 
user of accountability efforts is the general public, accountability 
reports, unlike assessment, are presented in summary form. Some 
states, including Colorado, have established standards of 
accountability, or specific acceptable levels of performance that 
institutions are required to meet. Florida, for example, has instituted 
a "rising junior" exam (which assesses whether college juniors have 
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gained an acceptable level of knowledge), and Colorado has developed 
statewide admission standards and ethnic minority graduation rates. 

Standards: Standards are predetermined levels of achievement that institutions or 

students are expected to meet. 

Postsecond^ry Postsecondary institutions include all 28 Colorado publicly- 
Institutions: supported two-year community colleges, and 

four-year colleges and universities. 



m. VALUES OF ASSESSMENT AND ACCOUNTABILITY 
The Value of Assessment 

There are several reasons why postsecondary institutions consider assessment activities 
valuable. The primary value of assessment to institutions is determining how effective 
instruction is for student learning. In other words, how much students are learning, and 
whether students are actually learning what they are expected to learn* Individual campuses 
use the results of assessment to make curriculum changes that improve instruction, and 
ultimately, student learning. The assessment process itself caa be a useful tool to ensure 
dialogue about curriculum both within and between academic areas. As a panel of Colorado 
assessment directors at a recent conference told the audience, "The assessment process can 
be as illuminating as the results." Assessment can also help to shift institutional focus from 
research to instruction. As another faculty member recently observed, "What gets evaluated 
is what gets noticed." 

The Value of Accountability 

Accountability is valuable from the statewide perspective in two ways. First, it ensures that 
the postsecondary system not only offers, but also delivers to individual citizens, the 
knowledge and skills that they need. In addition, accountability assures taxpayers can be 
confident that the postsecondary system which they provide is effective. As Howard R. 
Bowen stated in The Products of Higher Education , published in New Directions for 
Institutional Research . Volume 1, Spring, 1974: 

"The idea of accountability is quite simple. It means that colleges and 
universities are responsible for conducting their affairs so that the outcomes 
are worth the cost. It implies that institutional eflTorts should be directed 
toward appropriate goals and that the outcomes should be consistent with 
these goals and should be achieved at minimum cost. It also implies that the 
institution should report credible evidence on the degree to which it is 
achieving its mission and on its costs." 

4 
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IV. CURRENT ASSESSMENT AND ACCOUNTABILITY EFFORTS IN COLORADO 
Assessment Activities 

Colorado postsecondary assessment activities described in this paper range from the 
legislatively mandated Higher Education Accountability Program (HEAP), to institutional- 
based activities such as the ongoing review of existing academic programs* In addition, it 
is important to acknowledge a wide variety of on-going institutional efforts to improve the 
educational experience of students* 

1* COLORADO HIGHER EDUCATION ACCOUNTABILITY PROGRAM (HEAP) 

The Colorado Higher Education Accountability Program was established by the legislature 
in 1985 (C.R.S* 23-13-101). Among other specific directives, the law requires that 
institutions of higher education demonstrate improvements in student knowledge, capacities, 
and skills between entrance and graduation. Li addition, the law directs CCHE to report 
the status of institutional implementation efforts to the legislature annually. CCHE staff 
monitor HEAP at each institution, provide assistance to institutions, and review institutional 
compliance with the law. Specifically, CCHE staff look for evidence tiiat each institution 
assesses students in terms of their general education as well as their understanding of their 
chosen academic fleld. In addition, CCHE staff document changes that institutions make 
in curriculum and practice as a result of assessment flndings. 

Institutional HEAP reports reveal that institutions use a variety of techniques to measure 
student outcomes in both general education, academic Held, or both. Some institutions use 
standardized tests such as the ACT COMP (College Outcome Measures Program) to assess 
students' understanding of general education. More and more institutions are, however, 
replacing standardized tests with other techniques such as general education skills tests. 
These exams assess students' abilities to read, compose and compute. Many of these tests 
are developed by faculty at each campus who are familiar with individual curriculums nnd 
desired student outcomes. 

Institutions use a combination of several measures to assess students' grasp of their 
academic field. One frequently used approach is the Major Field Assessment Test (MEAT), 
a standardized test of academic fleld and emphasis area. Another approach is the use of 
portfolios, a accumulation of materials that students have produced throughout their 
education. Portfolios provide a subjective means for substantive faculty evaluation of 
whether or not the student has a good grasp of their fleld. The portfolio approach is often 
used together with a senior seminar course, which integrates learning from major classes. 
Assessment of actual performance is yet another type of assessment used mostly in the 
performing and visual arts, as well as in engineering. Students are assessed on their 
performance in recitals, performances, exhibitions, or demonstration projects. 
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Finally, institutions survey alumni, employers, and both continuing and former students 
to assess student opinions of knowledge attained in both general education and chosen 
academic field* 

2. INSTITUTIONAL ACADEMIC PROGRAM REVIEW 

State law passed in 1985 [C*R*S* 23-1-107 (3)] directed governing boards to periodically 
review and evaluate academic programs at each of their institutions, consistent with 
institutional role and mission* Although this law contamed the first mandated assessment 
measures in Colorado, institutional review of academic programs has been in place for 12 
years. Unlike HEAP, which focuses primarily on student outcomes, academic program 
review uses student outcomes as just one element of a comprehensive process to determine 
program effectiveness. Institutions examine both programmatic achievements and 
deflciencies to strengthen programs and ultimately student learning* Frequently, findings 
of academic program review result in curriculum changes. Some institv'tions couple 
academic program review to HEAP activities. 

Accountability Activities 

Accountability activities in Colorado range from CCHE analysis and presentation of 
statewide data about the performance of the state's postsecondary system to institutional 
and academic program accreditation review. Governing board staff also analyze 
accountability data. 

1. HIGHER EDUCATION ACCOUNTABILITY PROGRAM (HEAP) ANNUAL 
REPORTS 

The presentation of the HEAP annual report to the state legislature, and the dissemination 
of its findings, is one means of informing the public of the effectiveness of the public 
postsecondary system. The 1992 HEAP report described signiflcant progress of students 
attending Colorado colleges and receiving degrees, as well as serious concerns. It reported 
that student advising is deflcient at many institutions, and that remedial education, teacher 
education assessment, and minority retention and graduation rates should all be improved. 
While not highly publicized beyond the higher education community, this report did receive 
the attention of state legislators. 

2. INSTITUTIONAL AND ACADEMIC PROGRAM ACCREDITATION 

Regional accrediting associations, such as the North Central Association of Colleges and 
Universities, play an important role in both accountability and assessment by providing 
public assurance of an institution's effectiveness. Traditionally, accrediting associations 
have required institutions to demonstrate, once every Ave to seven years, that they have 
adequate human, financial, and physical resources necessary to be effective. Recently, 
however, accrediting associations added student outcomes as a significant component of 
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accreditation review* The North Central Association (Commission on Institutions of Higher 
Education) 1989 Statement on Assessment and Student Academic Achievement states: 

"The Commission wants to make clear that all institutions are expected to assess 
the achievement of their students* With this statement we make explicit the 
Commission's position that student achievement is a critical component in assessing 
overall institutional effectiveness* Our expectation is that an institution has and is 
able to describe a program by which it documents student academic achievement*" 

The North Central Association has accepted annual HEAP reports from some Colorado 
institutions to fulfill the expanded student assessment requirements* 

In addition to institutional accreditation by regional accrediting associations, academic 
programs also undergo accreditation by specialized accrediting bodies* Forty-two 
specialized accrediting bodies review academic programs at all levels to ensure adequate 
professional preparation of students* The National League for Nursing, for example, 
reviews associate through graduate level degree programs* Specialized accrediting bodies 
review a very wide range of disciplines, including business, education, interior design, law, 
and construction* 

3, ANNUAL SCORECARD ON COLORADO PUBLIC HIGHER EDUCATION 

The Scorecard is an annual report of the status of certain aspects of public postsecondary 
education in Colorado that is designed to provide educators, public policy makers, and the 
public a measurement of the performance of Colorado college students and the 
postsecondary system* The Scorecard presents eighteen measures to evaluate the system's 
effectiveness in four systemwide educational values* These values are educational 
excellence, educational access and diversity, efficient delivery of education, and adequacy 
of educational resources. 

4. OTHER CCHE PUBLISHED REPORTS 

CCHE produces other reports on a regular and periodic basis that provide the public with 
various measures of the performance of the state's postsecondary system* 

Persistence and Completion Rates^ Colorado Public Higher Education ; 

This report presents the Hndings of CCHE analysis of full-time student graduation, 
transfer, and persistence raves for all Colorado publicly supported two-year community 
colleges, and four year coU^es and universities* Data is provided for both in-state and out- 
of-state students, by age, ethnicity, and gender* This information is presented both by 
individual institution and summarized by two and four-year sectors, allowing interested 
persons examine and compare the performance of students at a specific institution, or 
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the entire system. This report was produced for the first tiiae In 1991. Annual editions 
are anticipated. 
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status of Diversity in Colorado Higher Education 



This annual report provides descriptive information on enroUment, degrees granted, and 
employment at Colorado publicly supported postsecondary institutions* Data is provided 
by student residency, student class load, degree level, age, ethnicity, and gender* This 
information is presented in summary form by type of institution, and by the entire system, 
allowing exammation and comparison of performance by groups of institutions, or by the 
system as a whole* Detailed data about the performance of individual institutions on these 
measures is contained in the Digest of Colorado Postsecondary Education Statistics , another 
CCHE document* 

Certificates and Degrees Conferred in Colorado Public Colleges and Universities 

This annual report provides descriptive information on the characteristics of students who 
received certificates and degrees from Colorado publicly supported postsecondary 
institutions during the previous five years* Data is provided by academic discipline, degree 
level, ethnicity, and gender* This information is presented by individual institution, type 
of institution, and by the entire system, allowing examination and comparison of 
performance by individual institution, type of institution, or by the system as a whole* 

5. CCHE POLICIES WTIH ACCOUNTABILITY COMPONENTS AND STANDARDS 

In addition to the annual CCHE Accountability Report, the Scorecard, and other CCHE 
accountability related reports, CCHE also maintains statewide policies that prescribe 
specific standards with which institutions are required to comply* CCHE examines and 
reports annually on the compliance of institutions with these policies* 

Ethnic Minority Graduation Goals; 

CCHE and the higher education governing boards have established a goal to increase the 
representation of ethnic minority students at all levels of postsecondary education* The goal 
is that by the year 2000, 18*6% of college graduates from Colorado postsecondary education 
institutions will be ethnic mmorities. (In 1988-89, 18*6% of Colorado high school graduates 
were ethnic minorities.) Under the program, each college sets its own annual goal for the 
percentage of rninorities who will graduate* The annual institutional goals must, however^ 
result in the attainment of the institution's goals by the year 2000, which in turn 
contributes to the statewide goal* CCEDB holds colleges accountable for these goals by 
comparing actual graduation rates with institutional goals, and then publicly announcing 
the results* Institutions that do not reach their annual goals are required by CCHE to 
develop plans to ensure future compliance with the policy* This policy provides a 
standard, or acceptable level of achievement that institutions are expected to meet* 
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Statewide Admission Standards: 



Another area in which CCHE requires a specific level of institutional achievement is the 
statewide admission standards policy. The admission standards, mandated by legislative 
directive in 1985 and first implemented by CCHE in 1986, require that four-year 
institutions offer admission primarily to those first-time freshmen who have demonstrated 
the capacity to succeed (either by high school performance or standardized test scores) at 
that institution. CCHE monitors institutional compliance with these standards, and assesses 
a financial penalty on institutions that do not maintain this level of performance. 



V. EXAMINING THE ADEQUACY OF CURRENT ASSESSMENT AND 
ACCOUNTABILITY EFFORTS 

Criteria for examination 

Several criteria are useful in determining the adequacy of both existing and potential new 
assessment and accountability efforts. Primary criteria include both the practical 
effectiveness and cost effectiveness of the effort. Effective assessment efforts produce 
feedback about student learning that enable faculty and the institution to improve its 
curriculum and teaching practices. Effective accountability efforts provide mformation to 
the public and key decision makers of the performance of the staters postsecondary 
education system, allowing them to develop informed views about the effectiveness and 
efficiency of the system that they pay for. While assessment and accountability efforts must 
be well designed to provide useful information, they must also be designed to provide 
maximum benefit for expenditures of both tune and money. 

Analysis of Current Assessment Efforts 

1. COLORADO HIGHER EDUCATION ACCOUNTABILITY PROGRAM (HEAP) 

The Colorado HEAP is a relatively new program in Colorado. Although it was established 
by the legislature in 1985, CCHE was given until 1990 to develop the program. Institutions 
began assessmg student performance in 1989. Therefore, a substantial evaluation of the 
program^s effectiveness after only three years of implementation may be premature. It is, 
however, possible to make some observations about the practical and cost effectiveness of 
efforts to date. When evaluating the effectiveness of HEAP as an assessment effort, it is 
important to keep in mind that the primary intended users of assessment are faculty. 

The Education Commission of the States has named Colorado as one of the national leaders 
in the development of an effective statewide accountability program* It is apparent that 
HEAP is becoming more and more useful to institutioios as it matures and becomes an 
established part of the campus culture. Institutional HEAP administrators report that 
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faculty accept assessment information more readily and make necessary curricular and 
teaching changes to improve student learning. Institutions also report Lncre^^ed value to 
administrators from their broader institutional perspective. 

CCH£ analysis of the costs of HEAP reveal that institutions estimate they spend at least 
$2.5 million on HEAP during FY 1990-91, down slightly from $3 million the previous year. 
Statewide, institutional HEAP expenditures averaged .6% of the total state appropriation 
and tuition revenue for that year. On an institutional basis, expenditures ranged from 
.01% (of state appropriation plus tuition) at Otero Junior College in 1990, to 1.51% at 
Pueblo Community College in 1991. Four-year institutions reduced their HEAP 
expenditures between the first and second years by 47%, while two-year schools increased 
their expenditures by 22%. The four-year institution decrease was in part due to initial 
computer hardware and software purchases at several of the larger institutions. In 
addition, several four-year institutions reduced their costs by discontinuing standardized 
tests. There is no doubt that statewide assessment programs require expenditure of scarce 
resources. The National Center for Higher Education Management Systems (NCHEMS) 
estimates the annual cost of one value-added assessment at approximately $29,000 for a 
smaU liberal arts college, and over $120,000 at a major public research university. CCHE 
and governing boards also incur costs associated with HEAP. Approximately 50% of one 
CCHE staff person's t^:ne is devoted to this effort. 

While most institutions recognize the internal value of HEAP assessment efforts, the 
program is not without criticism. Some institutions have expressed frustration about the 
cost of HEAP. Some institutions, especially those with severely constrained resources, 
would prefer to spend these precious resources on other activities. Other institutions do 
not view HEAP as an essential activity. Some institutions have also expressed concern about 
the HEAP annual reporting requirements, preferring to report less frequently, redirecting 
resources to assessment rather than reporting efforts. CCHE staff have responded to this 
suggestion by permitting 8 mstitutions that are making satisfactory progress to provide in- 
depth reports once every three years, instead of annually. To date, only four of the eight 
have elected to accept this alternative. The other four institutions consider th^ annual 
reports either intrinsically valuable to institutional review efforts, or a valuable element in 
their regional accreditation review. In addition, institutions have requested that HEAP be 
tied more closely to, or replaced in part by, the existing multi-year institutional 
accreditation process. Finally, some institutions charge that the HEAP annual reporting 
requirement of graduation and persistence data is redundant with other CCHE data 
reporting requirements. Contrary to this perceived redundancy, CCHE staff provide 
institutions that wish to do so the opportunity to review graduation and persistence data 
prior to release, and make necessary changes. Staff does not require institutions to submit 
the same data twice. 

In summary, while certain reporting aspects of HEAP could be reviewed for improvement, 
overall, HEAP appears to be satisfactorily achieving its objectives at this point in time. The 
program should continue to be monitored closely for continued effectiveness. 
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2. INSTITUTIONAL AND ACADEMIC PROGRAM ACCREDITATION 



Institutional accreditation by regional accrediting associations appears to be achieving 
accountability objectives* To the public, accreditation provides an assurance of acceptable 
quality* The effectiveness of institutional accreditation as an assessment effort is, however, 
less evident at this time, because student assessment requirements were only recently added 
to the institutional accreditation process* Institutional accreditation efforts should be 
monitored to ensure effectiveness in meeting assessment objectives* 

Program accreditation by specialized accrediting bodies appears to be effectively meeting 
accountability objectives* Most institutions aspire to achieve the status that accompanies 
specialized accreditation of programs provides, and by doing so, must meet standards 
specified by the profession* Like institutional accreditation, program accreditation provides 
quality assurance to the public* The effectiveness of specialized accreditation in meeting 
assessment objectives is less apparent, however, because assessment of student outcomes is 
not as required in specialized program review as it is in institutional program accreditation* 

3. INSTITUTIONAL ACADEMIC PROGRAM REVIEW 

Academic program review efforts appear to be very effective in assisting faculty, 
administrators, and governing board members in integrating their programs into statewide 
goals of quality, access and diversity* The external evaluation component of program 
review by professional societies and peer faculty from other institutions has provided 
valuable feedback for program improvements* Many positive programmatic changes can 
be attributed to the program review process* Governing boards have, for example, 
discontinued or consolidated programs that were no longer needed, thereby redirecting 
resources to evolving or high priority programs* Program review has also helped governing 
boards identify especially strong programs for exemplary recognition or advancement such 
as nomination as a CCHE Program of Excellence, or national grants* 

The only criticism of the effectiveness of acadenrc program review is from some legislators 
who observe correctly that the process does not provide data that is comparable between 
institutions* In response to this concern, it is important to note that academic program 
review, as an assessment activity * is designed for institutional self-assessment, not for inter- 
institutional comparisons* Other accountability activities provide information more 
appropriate for making comparisons among institutions* 

While it may be useful to know the statewide costs of academic program review ^ actual cost 
data is not readily available* Because this activity has been implemented for a number of 
years, the costs are integrated into ongoing institutional operations* 

In conclusion, academic program review is very effectively achieving its objectives at this 
point in time* 
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Analysis of Current Accountability Efforts 



Effective accountability efforts provide information to the public and key decision makers 
of the performance of the state's postsecondary education system, allowing them to develop 
informed views about the effectiveness of the system that they pay for. 

!• fflGHER EDUCATION ACCOUNTABILrrY PROGRAM ANNUAL REPORTS 

Each year CCHE produces an annual report on the status of higher education 
accountability efforts. Tbis report is one element of the broader HEAP program described 
on page 5 of this paper, llie first annual EDEAP report to the l^islature was published in 
1990. This report focused primarily on the extent to which institutions had developed 
mechanisms to implement HEAP in its first year. Although the second annual report, 
produced in 1991, contained signiflcant detail about institutional assessment results, this 
information was not comparable among institutions. As a result, several legislators 
expressed disappointment about the usefulness of the report from the legislative perspective. 
As a result of this concern, and the natural evolution of HEAT at the institutions, the third 
annual report, published in 1992. contained substantive information not only about student 
outcomes, but also concerning statewide policy implications of these findings. This 
legislative concern is the understandable result of apparent confusion between the 
expectations of accountability and the expectations of ass^ment in the HEAP legislation. 
The legislation specifically directs institutions to assess the knowledge, capacities, and skills 
of students, and to "continuously examine and adjust the content and delivery of its 
curriculum ..." This directive is clearly assessment oriented. At the same time, the 
legislation prescribes an accountability effort by directing institutions to communicate the 
results of assessment efforts to the public. The clash of expectations between these 
assessment and accountability directives is clear in the annual HEAP report produced by 
CCBDE. While the report presents the broad results of assessment, i.e„ insufficient student 
advising and retention, it does not provide information to allow college-by-college 
comparison of these factors. 

In summary, while HEAP is legislatively designed to be assessment oriented, it is also 
expected to provide accountability. These dual expectations of assessment and accountability 
pose a difficulty in determining the effectiveness of the HEAP annual reports. From an 
assessment perspective, the annt^al report effectively provides information about how 
institutions are addressing deficiencies identified through various assessment measures. It 
also reveals statewide policy implications for legislators and public policy makers to 
address. In this way, it achieves legislative objectives effectively. From the accountability 
perspective, however, the annual HEAP report does not provide information that permits 
institutional comparisons that some may have expected. Considerable dissatisfaction has 
been expressed about this perceived deficiency. From a cost-benefit pei*spective, the CCHE 
staff resources required to produce this report are moderate. 
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2. ANNUAL SCORECARD ON COLORADO PUBLIC HIGHER EDUCATION AND 
OTHER CCHE PUBLISHED PEPORTS 



The Scorecard could be considered the most comprehensive accountability-designed 
document produced by CCHE. While other CCHE publications provide more 
comprehensive detailed information about certain aspects of the performance of the state's 
postsecondary system, the Scorecard presents summary information about a number of 
measures in one document. Although the 1992 ScoreCard contains flndings for 18 
measures of these values, the only aspect of the report that was widely reported by the 
media throughout Colorado was adequacy of postsecondary funding. To a lesser extent, 
ethnic diversity of the postsecondary system also received some media attention. One 
legislative criticism of the first annual Scorecard which was produced in 1991, (and to some 
extent the 1992 edition as well), was that most of the measures did not report the 
performance of individual institutions on specific indicators. Instead, the information was 
summarized and reported by type of institution. Another criticism was that the S corecard 
did not reveal how institutions performed on prescribed standards, or predetermined, 
required levels of achievement. 

From a benefit perspective, although the CCHE staff resources required to produce this 
report are significant, this expenditure is valuable compared to the potential usefulness of 
the Scorecard . 

In summary, the Scorecard has strong potential for becoming a most valuable tool in the 
a CCHE statewide accountability effort, especially as valid concerns about the document 
are addressed. 

The effectiveness of other CCHE published reports, as accountability efforts, varies. The 
release of Persistence and Completion Rates. Colorado Public Higher Education , received 
very significant attention in the legislature and the media statewide. This report, which 
revealed that only 19% of 1986 first-time freshmen at four-year Colorado postsecondary 
institutions graduated within four years, led to the passage of two bills addressing higher 
education productivity. Senate bills 155 and 59. Similarly, Status of Diversity in Colorado 
Higher Education , which reported significant underrepresentation of ethnic minorities 
throughout Colorado postsecondary education, also received significant attention from state- 
level policy makers, legislators, and the media-all audiences that the report was intended 
to inform. Subsequent to the release of this report, CCHE strengthened the statewide 
Ethnic Minority Graduation Goal Policy to include financial consequences for non- 
compliance with specified performance standards. 

On the other hand, the release of Certificates and Degrees Conferred in Colorado Public 
Colleges and Universities , has only a moderate impact as an accountability effort. While 
this document currently serves a useful information role within the postsecondary education 
community, it does not receive the attention of audiences that accountability efforts are 
intended to reach. In part this phenomenon may be due to the timmg of when this 
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document is produced. It is released several months subsequent to the release of Status of 
Diversity... and CCHE postsecondary enrollment data, which contains similar conclusions. 

From a cost-benefit perspective, although the CCHE staff resources required to produce 
these reports are significant, this expenditure is valuable compared to the benefit received. 

In summary. Persistence and Completion Rates. Colorado Public Higher Education , and 
Status of Diversity in Colorado Higher Education are very effectively achieving their 
desired accountabiUty objectives. Certificates and Degrees Conferred in Colorado Public 
Colleges and Universities should either be reviewed for greater effectiveness as an 
accountability effort, or no longer be considered for its value regardmg accountability. 

3. CCHE POLICIES WITH ACCOUNTABILITY COMPONENTS AND STANDARDS 

As a statewide coordinating board, CCBE is in a unique position to hold governing boards 
and institutions accountable for achieving certain specified standards of performance. 
CCHE statewide admission standards, and ethnic minority graduation goals, for example 
specify performance levels that institutions must meet in order to avoid financial sanctions. 
These policies effectively achieve accountability objectives in several ways. Institutions and 
governing boards use these standards in their planning efforts. In addition, in order to 
avoid sanctions, institutions that fail to meet specified standards must explain to CCBE (in 
a public settmg) why past efforts have failed. Most importantly, institutions must also 
publicly present evidence of an effective strategy to achieve desired goals. This approach 
has proven effective with both admission standards and ethnic minority graduation goals. 



VI. NATIONAL PERSPECTIVES ON ASSESSMENT AND ACCOUNTABILITY 

As stated at the outset of this paper, the purpose of this project is to assist the Commission 
in determining the sufHciency of current academic assessment and accountability efforts in 
Colorado's postsecondary education system. Such an examination must consider related 
discussions of assessment and accountability occurring at the national level, and their 
appropriateness for Colorado. The most prominent consideration currently is the Task 
Force on Assessing National GosAs Relating to Postsecondary Education, established by the 
National Governors' Association (NGA) in February 1992. Before presenting the conclusions 
of the task force, it is important to recall the distinction between assessment objectives and 
accountability objectives as addressed in this paper. Assessment measures provide feedback 
on student outcomes that faculty use to enhance instruction. Accountability efforts inform 
the public of the performance of students in general. For the most part, accountability 
efforts, such as graduation rate reports, do not provide specific feedback that faculty can 
use to improve student learning. Although the objectives of assessment i?nd accountability 
differ, both are of value to an effective postsecondary system. 



ERIC 



15 

17 



In making its recommendations, the task force clearly distinguishes between its objective, 
"assuring public awareness and accountability, and the objective of stimulating reform via 
individually tailored assessments/* While not discounting the need for assessment efforts 
that institutions use to reform and improve student learning, the task force clearly views 
such efforts as beyond the scope of its NGA directive. 

Membership of the task force includes two college presidents and six state higher education 
executive officers, including David Longanecker, CCHE Executive Director, NGA directed 
the task force to: 

1) investigate and report on the feasibility, desirability and schedule for developing 
standardized comparable state reports on the rate at which students entering higher 
education institutions complete their degree programs and by minority status, and 

2) investigate and report on the feasibility and desirability of a sample-based 
collegiate assessment which would provide regular national and state representative 
indicators of college graduates' ability to think critically, communicate effectively 
and solve problems. 

The task force recommends developing a voluntary national system for states to report 
comparable data on graduation rates. In addition, tLe task foi ce recommends developing 
a voluntary national assessment system to provide national and state indicators of college 
graduates' ability to think critically, communicate effectively and solve problems. The task 
force believes that both of these efforts would be a valuable response to public calls for 
increased accountability. Referring to the task force recommendation for a system to 
report graduation rates, the task force report states: 

"This makes sense primarily because the consumers of postsecondary education 
want and deserve to know how likely it is that a student entering a given institu('lon 
will stay in the institution and ultimately graduate.** 

Similarly, the report provides the task force rationale for a voluntary national college 
graduate assessment system from a public accountability standpoint: 

**Today, no systematic way exists to evaluate the effectiveness of the system, either 
based on identified standards of performance or even on a comparative basis. 
Existing informal evaluations, therefore are often based on valid or invalid 
institutional reputations and public relations efforts. Without a systematic way to 
gauge postsecondary education's effectiveness, public perceptions are diverging. 

The task force identifies a number of significant issues to be overcome prior to the 
development of both of these efforts. For the graduation rate reporting system, these issues 
include **mutually agreed-upon persistence indicators** (**normal*' progress, for example), 
**meaningful degree completion statistics** (credit and programs completed, as well as 
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aggregate graduation rates), and other "contextual indicators" that assist the public in 
understanding student graduation statistics. Obstacles to developing a valid national 
assessment of college graduates, the task force noted, included the amount of time required, 
cost, as well as the difflculty of gaining consensus about the objectives of a postsecondary 
education. In addition, the task force report, noting the potential signiflcant reluctance of 
from within the postsecondary community states, "Because objective information will 
challenge the validity of the current informal hierarchy in postsecondary education, those 
who currently benefit may well challenge the usefulness of such a system." 

The report concludes by asserting that the development of national standards of 
performance and assessment will require national leadership and federal support. 



Vn. OPTIONS FOR COMMISSION ACTION 



In completing its directive to develop an analytic framework for CCHE Commissioners to 
examir<; the sufflciency of current academic assessment and accountability efforts in 
Colorado's postsecondary education system, the team assigned to this project presents the 
following four options for Commissioner consideration: 

Option 1: Maintain assessment and accountability efforts that are currently underway. 

Option 2: Discontinue (or downsize) current assessment activities, while enhancing 
accountability efforts. 

Option 3: Develop one system to accomplish objectives of both assessment and 
accountability. 

Option 4: Continue current assessment activities, whUe enhancing accountability efforts. 

Discussion of Option 1; Maintain Assessment and Accountability Efforts that are 

Currently Underway. 



One approach would be to maintain the assessment and accountability efforts that are 
currently in place. Although possible improvements have been suggested in current 
assessment activities, (Colorado HEAP, regional accreditation of academic programs, and 
institutional program review) for the most part these efforts seem to be valuable in 
determining the effectiveness of instruction on student learning, and on developing a 
capacity within institutions for self-evaluation. Another point in support of HEAP efforts 
is that initial investments of tune and effort are now beginning to pay off after four years 
of implementation. The institution's estimated cost of implementing HEAP has decreased. 
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Similarly, current accountability efforts, (HEAP Annual Reports, CCHE Scorecard and 
other published reports, and CCHE policies with accountability components and standards) 
perform a valuable role in informing the public and their elected representatives of the 
performance of the postsecondary system that they provide* 

Maintaining the status quo with regard to current assessment and accountability efforts, 
would not, however be responsive to the concerns of institutions that assessment activities 
(especially HEAP) are costly, redundant, and less than essential. Nor would this approach 
be responsive to legislators who assert that accountability efforts, particularly the content 
of annual CCHE HEAP reports and the Scorecard . could be improved to provide more 
information useful to legislators about the performance of the postsecondary system. 
Furthermore, efforts to effectively inform the public of the performance of the 
postsecondary system would not be advanced by the status quo approach. 

Discussion of Option 2; di scontinue (or downsize) Current Assessment Activities. WhUe 

Enhancing Accountability Efforts. 

Discontinuing or reducing the requirements of the Colorado HEAP, while at the same time 
enhancing accountability efforts, could be an alternative favored by those who would prefer 
less institutional autonomy and more statewide reporting of information. This alternative 
would be sensitive to institutional efforts to contain costs within a tight fiscal environment. 

Exchanging, in full or in part, accountability efforts for assessment efforts, which have very 
different objectives could send a message to institutions to redirect their efforts away from 
institutional review and improvement. This approach could seriously diminish institutional 
reform efforts in which institutions have invested a great deal and have already benefitted 
from. In addition, discontinuing or downsizing HEAP after only four years could make 
institutions less Ukely to take future statewide efforts seriously. Eliminating or significantly 
reducing HEAP would require legislative change. Finally, enhancing accountability efforts 
implies that effective alternate strategies are known. This is by no means certain. 

Discussion of Option 3; Develop One System to Accomplish Objectives of Both 

Assessment and Accountability. 

Another approach would be to develop one approach that accomplishes objectives of both 
assessment and accountability simultaneously. On one hand, this could be a more efficient 
approach, ideally. Given, however, the somewhat different objectives of assessment and 
accountability, the possibility of devising a strategy that produces useful information for 
all intended audiences seems remote, expensive, and unlikely to accomplish in a reasonable 
amount of time. There is also the very real possibility that the effort to devise one measure 
might provide a temptation to "contaminate the t.'st" by assessing only those areas that 
would result in favorable student outcomes. 
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Discussion of Option 4: Continue Current Assessment Activities. While Enhancing 

Accountability Efforto. 

A final proposed alternative is to continue current assessment activities, addressing valid 
institutional concerns about HEAP reporting and other requirements, and at the same tune, 
worldng to enhance accountability efforts, especially Colc.^do involvement in the NGA 
national accountability efforts. Those opposed to the national effort to report graduation 
rates and student performance data will object that preparation of the information 
envisioned by the NGA task force will only draw precious resources away from other more 
valuable institutional activities and not lead to real institutional reform. Alternatively, this 
approach could complement state oriented assessment efforts, since the NGA suggested 
approach would produce little in the way of feedback that institutions could use to improve 
instruction. If the NGA accountability-oriented approach materializes, an assessment- 
oriented approach such as HEAP would be an essential companion effort to balance the 
objectives of assessment and accountability. 
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